Method, system, and program for personal data management using content-based replication

ABSTRACT

The present invention provides systems and methods for efficient data storage, management, and back up. In particular, the present invention provides devices, software, and processes for efficient replication of data.

The present invention claims priority to U.S. provisional applicationSer. No. 60/497,938, filed Aug. 26, 2003, the disclosure of which inherein incorporated by reference in its entirety.

FIELD OF INVENTION

The present invention relates to the fields of data management, datastorage systems and software used to manage and protect against dataloss from unexpected computer system failure.

BACKGROUND

There are few commercially available systems or tools that providesimple and effective management, synchronization, and protection ofpersonal data on a mobile or stationary personal computer system. Onereason for this is that mobile computers are portable and do not lendthemselves to stationary backup solutions such as storage subsystems orserver-based backup solutions. Another reason is that personal data isoften user-determined, and do not lend themselves to automated backupmethods that protect data by individual storage device or file system.Third, whereas application programs for personal data management, datasynchronization, and data protection are readily available, they areusually designed for specific tasks such as backup, replication, or datamanagement, rather than an integrated data management application foroperation on a portable storage device. As a result, mobile andstationary computer users have few effective alternatives for personaldata management.

Simple data backup programs on portable devices can be implementedeasily, but these systems are also inefficient in their use of storagespace, and require either larger capacities or removable media toprovide ongoing data protection.

Alternatively, protecting data with system-level backup or replicationmethods is complex as well as time-consuming, since most are designedfor operation on a large-scale computer or network. Moreover, thesemethods often depend on computer infrastructure such as operating systemand file system, making them difficult to install and operate for userswho are not adept in management and maintenance of computer systems.Storage-conserving backup and remote replication techniques such assnapshots make near-continuous data protection possible, but requiresystem-level software or file system to be installed on the dataprotection system.

Network backup methods, whether achieved over local-area or wide-areanetworks, are also problematic. Backup or synchronization using theseconnections to a remote server require initialization and authenticationbefore the operations can be initiated, adding time and inconvenience,while also restricting mobility. Furthermore, when such operations areconducted over a wide area network with a mobile computer, connectionbandwidth is often low, resulting in slow backup and synchronization.

Faced with these impediments, the personal computer user often performspersonal data backup and synchronization manually, i.e. on afile-by-file basis, onto a locally attached disk drive or removablemedium such as writeable CD, floppy disk, or flash-memory device. Theresult is that personal data protection is an ad hoc process limited toa few selected files, exposing many computer users to potentiallycatastrophic loss of their data.

SUMMARY OF THE INVENTION

The present invention provides systems and methods for efficient datastorage, management, and back up. In particular, the present inventionprovides devices, software, and processes for efficient replication ofdata.

In some embodiments, the present invention provides one or more of:

-   -   1. Personal data management and data protection implemented with        a computer program installed on a portable storage device.    -   2. Ability to operate independent of operating system or file        system.    -   3. An integrated system including a combined data management,        automatic backup, data snapshots, and data compression program        installed on a portable storage device.    -   4. A system designed for small-scale rather than large-scale        data protection.    -   5. Automated cataloging and backup of data that is initiated        upon external connection of a portable storage device. The        personal data management system (PDMS) system combines the        automatic and operationally transparent capabilities of Java and        external peripheral interfaces such as USB or Serial ATA and        provides automatic backup or data synchronization.    -   6. Use of fingerprints to create a unique file block identifier        from block-level checksum, message digest, and other metadata        such as time stamp, file name, application source, size, and        file extension.    -   7. Data management and data protection based on combined usage        of file catalog, fingerprint engine, file indices, fingerprint        database, data files, and data compression engines.    -   8. Use of a combination message-digest and checksum algorithm to        effect incremental backup, or snapshot, of personal data files        through mathematical reduction of their contents. In additional        embodiments, the invention provides one or more of the following        features:        -   a. Detection of file changes by dividing the file into data            blocks and applying a checksum function to compare each            block against a database of previously stored blocks.        -   b. Byte-by-byte application of a checksum function            sequentially through data blocks comprising each file to            identify the start and stop points of the data change.        -   c. Verification of block-level changes through comparison of            message digests for stored data blocks.    -   9. Use of file block contents rather than a file system block        map changes to create a data snapshot.    -   10. Improvement of snapshot processing efficiency through the        application of checksum function to identify block changes prior        to use of computationally intensive message digest function to        verify the changes.    -   11. Improvement of snapshot processing efficiency through        comparison of message digests of block contents rather than the        actual block data.    -   12. Reduction of required data storage capacity through        retention of content-based data snapshots rather than addition        of new files.    -   13. Improvement of compression efficiencies through selection of        compression technique matched optimally with file-name extension        and associated data type.    -   14. A computer program that automatically clears portable device        storage space for data snapshots as device storage limits are        reached. In additional embodiments, the invention provides:        -   a. User option for snapshot overflow to a second computer or            storage device when user-specified storage limits are            reached, transferring older snapshot data on a            first-in-last-out basis until sufficient storage space is            created.    -   15. Synchronization and retention of catalog and fingerprint        data when data files are transferred to a second computer,        providing a means by which to verify the integrity of offloaded        data files based on message digests rather than actual data.    -   16. Ease of use through a pre-installed, platform-independent        computer program that initiates backup automatically upon        insertion of a portable storage device, and continues to backup        data incrementally on succeeding connections until user-defined        storage limits are reached.    -   17. Replication of data files based on a content extraction        method that enables present and prior versions of the data files        to be efficiently stored and retrieved from an application        program.

DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a diagram of an exemplary personal data management systemdevice of the present invention configured to operate with a desktopcomputer.

FIG. 2 shows an exemplary process for generating a fingerprint of thepresent invention.

FIG. 3 shows an exemplary data synchronization process of the presentinvention.

FIG. 4 shows an exemplary data management and storage process involvingmultiple computers.

DEFINITIONS

To facilitate an understanding of the present invention, a number ofterms and phrases are defined below:

As used herein, the terms “computer memory” and “computer memory device”refer to any storage media readable by a computer processor. Examples ofcomputer memory include, but are not limited to, RAM, ROM, computerchips, digital video disc (DVDs), compact discs (CDs), hard disk drives(HDD), and magnetic tape.

As used herein, the term “computer readable medium” refers to any deviceor system for storing and providing information (e.g., data andinstructions) to a computer processor. Examples of computer readablemedia include, but are not limited to, DVDs, CDs, hard disk drives,magnetic tape and servers for streaming media over networks.

As used herein, the terms “processor” and “central processing unit” or“CPU” are used interchangeably and refer to a device that is able toread a program from a computer memory (e.g., ROM or other computermemory) and perform a set of steps according to the program.

As used herein the term “encode” refers to the process of converting onetype of information or signal into a different type of information orsignal to, for example, facilitate the transmission and/orinterpretability of the information or signal. For example, audio soundwaves can be converted into (i.e., encoded into) electrical or digitalinformation. Likewise, light patterns can be converted into electricalor digital information that provides and encoded video capture of thelight patterns.

As used herein, the term “client-server” refers to a model ofinteraction in a distributed system in which a program at one site sendsa request to a program at another site and waits for a response. Therequesting program is called the “client,” and the program that respondsto the request is called the “server.” In the context of the World WideWeb, the client is a “Web browser” (or simply “browser”) that runs on acomputer of a user; the program which responds to browser requests byserving Web pages is commonly referred to as a “Web server.”

As used herein, the term “Internet” refers to any collection of networksusing standard protocols. For example, the term includes a collection ofinterconnected (public and/or private) networks that are linked togetherby a set of standard protocols (such as TCP/IP, HTTP, and FTP) to form aglobal, distributed network. While this term is intended to refer towhat is now commonly known as the Internet, it is also intended toencompass variations that may be made in the future, including changesand additions to existing standard protocols or integration with othermedia (e.g., television, radio, etc). The term is also intended toencompass non-public networks such as private (e.g., corporate)Intranets.

As used herein the term “security protocol” refers to an electronicsecurity system (e.g., hardware and/or software) to limit access toprocessor to specific users authorized to access the processor. Forexample, a security protocol may comprise a software program that locksout one or more functions of a processor until an appropriate passwordis entered.

As used herein the term “resource manager” refers to a system thatoptimizes the performance of a processor or another system. For examplea resource manager may be configured to monitor the performance of aprocessor or software application and manage data and processorallocation, perform component failure recoveries, optimize the receiptand transmission of data, and the like. In some embodiments, theresource manager comprises a software program provided on a computersystem of the present invention.

As used herein the term “in electronic communication” refers toelectrical devices (e.g., computers, processors, communicationsequipment, etc.) that are configured to communicate with one anotherthrough direct or indirect signaling. For example, a computer configuredto transmit (e.g., through cables, wires, infrared signals, telephonelines, satellite, etc) information to another computer or device, is inelectronic communication with the other computer or device.

As used herein the term “transmitting” refers to the movement ofinformation (e.g., data) from one location to another (e.g., from onedevice to another) using any suitable means.

As used herein the term “checksum” refers to a count of the number ofbits in a transmission unit that is included with the unit so that thereceiver can check to see whether the same number of bits arrived. Ifthe counts match, it's assumed that the complete transmission wasreceived. For example, both TCP (Transmission Control Protocol) and UDP(User Datagram Protocol) communication layers provide a checksum countand verification as one of their services.

As used herein the term “message digest” refers to the representation ofa block of data in the form of a single string of digits, created, forexample, using a formula called a one-way hash function. A hash functiontakes a long string (or message) of any length as input and produces afixed length string as output. The output is sometimes termed a “messagedigest” or “digital fingerprint.” Any change to the message, even of asingle bit, will (in all probability) result in a dramatically differentmessage digest when re-generated from the received message.

As used herein, the term “fingerprint file” refers to a file thatcontains data blocks of backup data, with associated fingerprint values.

As used herein, the term “directory snapshot file” refers to a file thatcontains the structure of the directory at the time of the snapshot. Foreach snapshot taken, an entry is made to the directory snapshot file toreflect incremental changes from the previous snapshot.

As used herein, the term “indices file” refers to a file that containsmapping between the data block signatures and the data block location inthe fingerprint file. New indices are recorded sequentially as snapshotsare taken.

As used herein, the term “mapping table” refers to a table that assignsand tracks the data block signature and the offset in the fingerprintfile. Preferably, it is structured to optimize the addition andretrieval of fingerprints.

As used herein, the term “last backup log” refers to a file thatcontains: (i) a list of all the files that existed in the system (thedirectory) at the time of last snapshot; and (ii) a memory pointer tothe associated indices file.

DESCRIPTION OF THE INVENTION

The present invention provides a personal data management system thatcatalogs, stores and protects data from hard disk failure on a mobile orstationary computer, using a computer program and a portable, externallyattached, data storage device such as USB flash memory or hard diskdrive. In preferred embodiments, the program uses a combinationmessage-digest and check-function based algorithm to manage and protectdata independent of specific computer operating system or file system,thus providing application-level portability for the personal datamanagement system. In preferred embodiments, this program ispre-installed on the portable storage device and is written in aplatform-independent language such as Java to provide interoperabilitywith different operating systems. In preferred embodiments, the systemcatalogs and manages data from a single portable device.

The personal data management system of the present invention, inpreferred embodiments, catalogs, stores and protects data from hard diskfailure on a mobile or stationary computer, using a computer program anda portable, externally attached, data storage device such as USB flashmemory or hard disk drive. In some such embodiments, the program uses acombination message-digest and check-function based algorithm to manageand protect data independent of specific computer operating system orfile system, thus providing application-level portability for thepersonal data management system. In preferred embodiments, this programis pre-installed on the portable storage device and is written in aplatform-independent language such as Java to provide interoperabilitywith different operating systems. Preferably, the system catalogs andmanages data from a single portable device, even though the data ofinterest may reside on more than one computer system or memory device.With this system of hardware and software, a complete and verifiablelisting of the data and part or all of the data itself can be kept inpersonal possession or in a secure place, while storing a duplicate setof data elsewhere.

Applications include, but are not limited to portable data back up;management, distribution, and synchronization of personal data betweentwo physically remote computer systems; version-based compression forapplication programs with repetitive data structure output such asdocument management, spreadsheet, and database programs; and transfer,management, retention and verification of important personal recordssuch as medical history, legal records, and contracts.

In some embodiments, the invention comprises a personal data managementsystem (PDMS) in which data files on a mobile or stationary computer canbe cataloged, searched, stored, and protected from hard disk failure,using a computer program and a single portable, externally attached,data storage device such as flash memory or hard disk drive. Anexemplary system is diagrammed in FIG. 1. For the purposes of thisdescription, personal data is defined as an ad hoc set of data filesdeemed to be of interest to a computer user. The program of the presentinvention is preferably installed on the portable storage device and iswritten in a platform-independent language such as Java so that it canbe operated on computers with different operating systems using a singledata management system. The system automatically backs up, synchronizes,compresses, and catalogs personal data files on a single device, eventhough the files of interest may reside on more than one computersystem.

In some embodiments, the PDMS computer program comprises one or more of:(a) a catalog containing stored files, snapshot and file metadatacontaining information such as time, date, file size, application sourcebased on file-name extension; (b) compressed and uncompressed user datafiles; (c) a fingerprint engine that generates a unique set ofattributes for each data file based on, for example, data contents,message digest algorithm, and a checksum function; (d) file indices thatpoint to, for example, offset, size, checksum, and message digest data;(e) a fingerprint database containing message digests and changes madeto the user data called snapshots.

The PDMS program, combined with a portable storage device with externalattachment interface, provides a single, integral system for managing,accessing, verifying, and protecting personal data against failure of acomputer or storage system. Applications include, but are not limitedto, personal data back up, management and transfer of personal databetween two physically remote computer systems, and transfer,management, and retention of important records such as medical history,legal records, and contracts.

In some embodiments, the PDMS duplicates or synchronizes data by firstcopying the data files of interest from a computer's hard disk, and thenupdating these files with data snapshots when the PDMS program isinvoked and changes are detected in data files. Data snapshots reducebackup time and storage space by backing up only those portions of filesthat have changed or wholly new files, rather backing up all fileswhether they have been changed or not.

In some embodiments, to generate snapshots, the PDMS software detectschanges in a set of personal files using a checksum and a message digestalgorithm. Changes are captured and stored in the form of acontent-based snapshot that comprises new data blocks and block-levelfingerprints derived from the data contents of each block. Here, themessage digest function is used to translate block data of arbitrarylength into an integer of fixed length, which is unique to the blockcontents.

In some embodiments, to detect changes, the PDMS (as exemplified in FIG.2) creates block-level fingerprints for each file by, for example: (a)dividing new files into fixed length blocks; (b) applying a checksumalgorithm for each new block against stored fingerprint datablock-by-block to determine whether the file blocks are new or where achange between old and modified files first occurs; (c) upon detecting achanged block, applying the checksum function on a byte-by-byte basis tolocate the points at which the file change begins and ends; (d) applyingthe message digest function to the changed bytes and then recording themessage digest and block checksum value in the fingerprint database; (e)when block fingerprints are found to match those already in thefingerprint database, block data is assumed to be the same, and pointersto the common fingerprint data are created; (f) repeating steps (b)-(e)until end of file is reached. Collectively, the message digests andchecksums comprise the fingerprints of files in each snapshot. Becausethese fingerprints are based on data content of each file, they provideidentification advantage when files are shared and stored among multiplesystems. In such instances, the message-digest based fingerprints offera superior identification method over file metadata such as time anddate stamps, because computer system clocks, especially those ondifferent networks, may differ.

In some embodiments, the PDMS message digest function is achieved withuse of a public domain, message-digest algorithm known as MD5 (“The MD5Message-Digest Algorithm”, R. Rivest, Network Working Group, Request forComments: 1321. MIT Laboratory for Computer Science and RSA DataSecurity, Inc. April, 1992, herein incorporated by reference in itsentirety), although other message digest algorithms may also be used.The MD5, algorithm takes message content of arbitrary length and createsa 128-bit “fingerprint” or message digest of the file content. Accordingto mathematical proof, each digest has a finite, but extremely smallchance of creating a duplicate digest for different messages.

In contrast with a conventional file system, the PDMS is an applicationprogram that can replicate data by: (a) initiating a snapshot; (b)identifying new data; (c) creating content fingerprints based on newdata; (d) creating a content-based snapshot from new data and pointersto previously stored data; (e) recording the fingerprint data with filemetadata in the system catalog; and (f) storing these data as asnapshot. These characteristics result in snapshots that are independentfrom the underlying system data structure. Consequently, moving datafrom or accessing the data on the portable device requires only the PDMSprogram, which, unlike a file system, installs easily onto computersystem. As an application program, the PDMS can be loaded dynamicallyinto user space memory, and requires no permanently assigned memoryspace. As a result, it is portable and may be installed on any computersystem that supports the PDMS application language.

In preferred embodiments, the PDMS provides integrated, multiple datamanagement functions based on its snapshot and fingerprint architecture.These functions include, for example: (a) creating backup or redundantcopies of data using a message-digest algorithm; (b) restoring olderversion files that are retained as snapshots; (c) reducing storagerequirements with data snapshots that eliminate repetitive data; (d)applying data compression technologies based on data types; (e)cataloging stored files and snapshot contents; and (f) providing asymbolic and unique representation of stored data, making it possible tokeep a verifiable record of the data on a local storage device withoutthe data itself. This symbolic representation capability not onlycreates a means by which to extend storage from the portable device ontoseparate system, but also enables the user to verify the integrity offiles that are catalogued on a local device and stored in complete formon a remote computer.

Snapshots and the catalog provide a history of changes made to files andlogic by which to reconstruct successive versions of the files,effectively reducing the storage requirements through versioning. Oncecreated, the snapshots and data changes are stored and optionallycompressed using any one of several standard data compressiontechniques. The specific technique is chosen based on the specific datatype, as indicated by file extension such as “.txt” or “.jpeg.” Usingthe file extension, an optimal method of compression can be chosen, orsometimes omitted, depending on the effectiveness of snapshots with thecompression method. This technique is based on prior work that has shownthat data repetition patterns and optimal compression techniques varywith data types (“Feasibility of Data Compression by EliminatingRepeated Data in Practical File Systems. Calicrates Policroniades andIan Pratt. Computer Laboratory, Cambridge University, Cambridge, UK, CB30FD. c 2002, herein incorporated by reference in its entirety).

In some embodiments, in instances where data storage requirements exceedthe limits of the portable device, the personal data management systemsoftware extends the data file storage to the hard disk of a secondcomputer system as exemplified in FIG. 3. In some embodiments, thesoftware achieves this by clearing a sufficient number of the oldestdata files from the portable storage device and transferring them to asecond computer to ensure availability of a minimum of storage space fornewly added data. The software then synchronizes the catalogs andfingerprint file of the portable storage with the second computersystem, creating a single, comprehensive catalog and fingerprint filesand attribute files for all personal data on the portable storagedevice, even though not all data files may be present on the device.Optionally, a password that corresponds to one located on the portablestorage device may be embedded in the second computer system's PDMSsoftware to ensure that access can be achieved only when the portabledevice is attached or access is specifically authorized from the firstcomputer system. Access to the catalog, fingerprint database and datafiles on the portable device may be password-protected, therebyproviding a secure and physically removable data repository that may bekept in personal possession or in a secure place.

The method for synchronizing storage using externally attached storageis the same whether data transfer is effected through intermittentconnection of a portable storage device with a first and then, secondcomputer, or through continuous device attachment and network connectionbetween the two computers as exemplified in FIG. 4.

In both cases: (a) software causes personal data to be duplicatedbetween a first computer and its attached storage device, therebyproviding a redundant copy to protect the data from device failure; (b)when the capacity of the attached storage device is exceeded, data iscopied and periodically synchronized with a second computer ofsufficient capacity to store all data. Under both intermittent andcontinuous network attachment instances, the externally attached storagedevice provides a catalog and fingerprint files for all personal datafiles, while storing all or part of the personal data on the device.When only part of the data is stored on the external storage device, asecond, complete copy of the personal is then maintained andsynchronized with that of a second computer.

In some embodiments, the system is used for large-scale backups. Forexample, in some embodiments, where the basic PDMS application is usedfor backup of large-scale storage devices and systems, the time requiredto complete a snapshot directly is dependent on memory-mappingefficiencies and main memory size of the attached computer system. Forlarge backups, fingerprint file size increases in proportion to backupsize. Thus, as fingerprint file size increases, virtual memory usagealso increases, extending the time required to complete a snapshotoperation.

To accommodate large-scale backups without large degradation of snapshotperformance, in some embodiments, the PDMS snapshot processing method ismodified to move memory-to-disk management from the computer system toinside the PDMS application itself. Under this modified method,fingerprint checking is limited to fingerprints brought into a memoryallotment of fixed size. To optimize performance, the resultantfingerprint information is later reconciled with the totality ofsnapshot information by a separate process, in which the snapshotdirectory file and indices are updated. While snapshot datareconciliation requires an additional step to the snapshot process, theincreased snapshot performance offsets the requirement forreconciliation and is not inconsistent with backup techniques used formanagement of large-scale backups.

The snapshot processing steps for large-scale snapshots in someembodiments of the present invention are described below:

-   -   1. The file name of the Indices File is recorded for each entry        made in the Directory Snapshot File.    -   2. In addition, recording of all directories and files on the        system when the last snapshot was taken along with a pointer to        information location contained in the Directory Snapshot File.    -   3. When a snapshot backup is made for block information not        already contained in the PDMS files, a related Indices File is        found based on its name in the Directory Snapshot File and        loaded into memory or a new one is created and backup is then        performed.    -   4. Each time new block information is encountered during a        snapshot, the Indices File is first checked against the Indices        File currently in memory, and if not relevant to the new        information, a new Indices File is loaded into memory.    -   5. In the event a single mapping exceeds the maximum permissible        size assigned to the Mapping Table, a new entry is made in the        Directory Snapshot File and a new Indices File is created. This        occurs when a single large file requires more memory than        allotted for the Mapping Table maximum. In such case, multiple        entries in the Directory Snapshot File are made, but all will        point to the same file with different offset.    -   6. To improve performance and eliminate redundancies created by        this internal memory management process, an independent        reconciliation process is periodically applied to Fingerprint        File. Upon identification of redundancies, the reconciliation        process removes the redundancy and updates the indices files        accordingly. The process also changes the values and names in        the Directory Snapshot File. Changes in the Directory Snapshot        files are performed following updates to the Indices File. The        reconciliation process is designed to run at times other than        when snapshot backup is being performed.    -   7. In the event an Indices File cannot be located, the PDMS        system can recover them by using the Fingerprint File and the        Directory Snapshot File and generating a new Indices File.        Recovery is achieved by locating the first entry that includes        the missing Indices File name, creation of the file, and        generation of the Indices File described in Directory Snapshot        File.

In some embodiments, this application-based memory management method forsnapshots includes the following processing rules and heuristics:

-   -   1. The Mapping Table is memory-resident, while the Indices File        is maintained on disk. A heuristic rule sets the Mapping Table        at two times the size of an Index File.    -   2. The Index File name is composed of the highest offset in the        Fingerprint File at the time the Index File when the first index        is added to the Index File.    -   3. For each entry made in the Directory Snapshot File, an        Indices File is created that contains name of the Indices File        used when the snapshot for that entry is taken.    -   4. Each directory or file that is backed up by the PDMS program        checks the previously used Indices File name and loads this into        memory. If memory limitations make it necessary, the previous        Indices File is written to disk and removed from memory.    -   5. With each snapshot backup taken, the PDMS system modifies the        Last Backup Log containing pointers to snapshot files within the        Directory Snapshot File. Each snapshot entry contains a pointer        to directory structure information within the Directory Snapshot        File. If a snapshot entry is made for directory information        already contained in the Directory Snapshot File, the new        snapshot entry incorporates the existing information. If new        snapshot entry information is not already contained in the        Directory Snapshot File, new directory entry information. This        entry information is generated by searching the Directory        Snapshot File, identifying all recent entries, and adding these        to a map that describes the new information and previous        snapshot directories/files and with corresponding Indices Files.        A snapshot entry thus constructed reduces snapshot        initialization time and eliminates need to scan the Directory        Snapshot file.    -   6. To maintain memory management within the PDMS application,        the Mapping Table that maps data block to finger print values is        limited to a fixed size assigned by the application.    -   7. Multiple indices mapping files are used where each file is        limited to one-half the size of the fixed size specified for the        Mapping Table.    -   8. When a Directory Snapshot entry is loaded into memory, the        PDMS application finds the matching Indices File by using its        name, which is composed of the, offset in the Fingerprint File        written in the Directory Snapshot File entry. The corresponding        Indices File is determined by selecting the file containing        whose name contains the highest value yet is lower than the        offset in the entry.    -   9. On initialization of the program, the Mapping Table starts        with its maximum set at half the maximum memory for Mapping        Table. As items are added dynamically, the Table is allowed to        grow to its maximum fixed size. Each Indices File is represented        by a different offset in the Fingerprint File and is set at        one-half the maximum Mapping Table memory size. When the Mapping        Table reaches its maximum allowable size, the entire Mapping        Table flushed or written to disk in two halves. Each half is        flushed with corresponding Indices File, which covers half of        the indices. By convention, the lower half is then freed from        memory, while the upper half remains in memory as the resident        Mapping Table.

In some embodiments, the systems and methods of the present inventionfind use in data recovery following unintended file loss. Applyingfingerprint and snapshot technology, the PDMS thus provides astorage-efficient system for computer disaster recovery, i.e., recoveryof a computer system the event of disk system failure. When configuredfor disaster recovery, the PDMS is preferably programmed to restore thecomputer system to its original operating condition by re-installing twotypes of data by different means: non-unique data and unique data.Non-unique data files include computer programs such as operatingsystem, application programs, and device drivers; most programs arecommercial products are typically distributed and maintained onnon-volatile storage media. Unique data files comprise user data frominput and program output, and configuration data from system registriesand configuration information for each application program; these dataare unique and cannot be replicated except from backup copy on awriteable storage device.

Using the PDMS, non-unique data is restored by copying from a referencesource such as commercial CD or authorized copy on magnetic disk. Incontrast, unique data is restored from the most recent PDMS snapshots ofuser data and configuration files.

Under the preferred PDMS method of backup and disaster recovery,fingerprints are first created for commonly used commercial computerprograms such as operating systems, word processing, spreadsheet, andpresentation applications. These fingerprints are incorporated into thePDMS program, which may be updated periodically as new program versionsare created and new fingerprints are created. In addition, the PDMS isprogrammed to recognize registry data and configuration data by filename, enabling the PDMS program to differentiate between configurationand user data.

To protect against disk system failure, the PDMS is used to takesnapshots of all data on the computer system to be protected. Programmedaccordingly, the PDMS snapshots avoid storage of non-unique data filesfor which it has fingerprints, thereby significantly reducing backupstorage requirements. Recognition of registry and configuration dataenables the PDMS program to separate storage of this data from user datawhen data snapshots are taken. When comprehensive restoration of data ordisaster recovery is required for a specific computer system, the PDMSidentifies all non-unique data files and either requests restoration bythe computer operator or automatically accesses a designated PDMSInternet site and downloads the associated data files. Following this,the PDMS restores registry and configuration data to restore operatingand application programs, and then installs user data to effect completerestoration of the computer.

All publications and patents mentioned in the above specification areherein incorporated by reference. Various modifications and variationsof the described method and system of the invention will be apparent tothose skilled in the art without departing from the scope and spirit ofthe invention. Although the invention has been described in connectionwith specific preferred embodiments, it should be understood that theinvention as claimed should not be unduly limited to such specificembodiments. Indeed, various modifications of the described modes forcarrying out the invention that are obvious to those skilled in therelevant fields are intended to be within the scope of the presentinvention.

1. A system for managing data, said system comprising a processor, saidprocessor configured to: a) receive data from a data source to bemanaged; b) generate a data block from said data; c) generate a checksumof said data block; d) identify whether said checksum has changedcompared to a prior analysis of said data source; e) if the checksum hasnot changed, generate a message digest of said data block; f) identifywhether said message digest has changed compared to a prior analysis ofsaid data source; g) if it is determined that a data block is changedafter conducting function d) or f), catalog the data and generate apointer to said data; and h) if it is determined that a data block hasnot changed after conducting function d) or f), catalog only a pointerto said data.
 2. The system of claim 1, wherein said cataloging offunctions g) and h) occurs without storing an entire copy of said data.3. The system of claim 1, wherein software that enables said processorto have functions a)-h) is contained in a portable device.
 4. The systemof claim 3, wherein said device weighs less than one pound.
 5. Thesystem of claim 3, wherein said device weighs less than one ounce. 6.The system of claim 3, wherein said software is configured to functionwith multiple different operating or file systems.
 7. The system ofclaim 1, wherein said processor is further configured to conduct one ormore of functions a)-h) in an automated fashion.
 8. The system of claim3, wherein said processor is activated to carry out one or more offunctions a)-h) when said portable device is connected to a computercontaining said data.
 9. The system of claim 1, wherein said processoris further configured to generate metadata for said data block, saidmetadata selected from the group consisting of time stamp, file name,application source, size, and file extension.
 10. The system of claim 1,wherein said processor is further configured to compress said data. 11.The system of claim 1, wherein function d) comprises detection of filechanges of a plurality of said data blocks by applying said checksumfunction to compare each block against a database of previously storedblocks associated with said data.
 12. The system of claim 11, whereinfunction d) further comprises application of said checksum functionsequentially through data blocks to identify start and stop points ofany identified data changes.
 13. The system of claim 12, whereinfunction d) further comprises verification of block-level changesthrough comparison of message digests for stored data blocks.
 14. Thesystem of claim 1, wherein said processor is configured to carry outfunction d) prior to function f), wherein function f) is not conductedif changes are identified in the use of function d).
 15. The system ofclaim 3, wherein said software is configured to store information insaid catalogue on two more separate computers.
 16. A method of datamanagement comprising the steps of: a) providing the system of claim 1;and b) interrogating data to be managed by conducting functions a)-h)with said processor.